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Abstract: The goal of this study was to determine if on-line out-of-class quizzing 
would lead to increases in (a) classroom engagement (b) academic performance 
and (c) preparation perception of college students. Twenty-four sophomore level 
students enrolled in a required functional anatomy course participated in this 
study. Results from this study indicate that on-line quizzing had a statistically 
significant impact on some indicators of student classroom engagement. 
However, on-line quizzing had no statistically significant impact on academic 
performance as evidenced by test scores. Finally, student perceptions, as a result 
of on-line quizzing significantly improved. 

I. Introduction. 

While research on in-class quizzing and student academic indicators is moderately 
represented in the literature, research regarding out-of-class on-line quizzing and student 
academic indicators is under represented. The authors in this study have utilized, at some point, 
traditional in-class quizzing in their courses, yet, none have trialed out-of-class on-line quizzing. 
Anecdotally, the measured outcomes from using in-class quizzes have varied among the authors, 
but, it is agreed that in-class quizzes can be somewhat cumbersome. 

For example, in-class quizzes take away from instructional time (anywhere from 10-20% 
of the allotted class), in-class quizzing adds a layer of class management by taking up 
instructional time so one can distribute and collect the quizzes. Furthermore, this time needed to 
take and manage the quiz also decreases the opportunities for students to directly engage the 
instructor on class topics and conversation and, visa versa, for the instructor to engage the 
students directly. 

When combining our collective experiences regarding in-class quizzing with the 
noticeable lack of research focusing on out-of-class on-line quizzing, we felt a study should be 
completed. This research need was further justified based on how uncertain the current literature 
is with regard to the effect quizzing has on various student academic indicators. We 
hypothesized that the impact of out-of-class on-line quizzing on student academic indicators 
would yield more unequivocal findings than what is currently documented on in-class quizzing. 
Thus, the present study was initiated. The hypotheses we tested were: 

1 . Out-of-class on-line quizzes would lead to an increase in student classroom 
engagement. 

2. Out-of-class on-line quizzes would lead to an increase in academic performance. 


1 Assistant Professor of Physical Education, murtell@iupui.edu . 

2 Associate Professor. 

3 Professor. 

4 Associate Professor. 

5 Professor, the above at Indiana University Purdue University Indianapolis, 901 West New York Street, Indianapolis, IN 46202. 


Urtel, M. G., Bahamonde, R. E., Mikesky, A. E., Udry, E. M., and Vessely, J. S. 


3. Out-of-class on-line quizzes would lead to an increase in student perception. 

II. Review of Related Literature. 

The concept of quizzing in higher education is not novel. In fact, quizzing has been used 
by the faculty of college classrooms as an assessment tool, a teaching technique, or some 
combination for over thirty years (Mawhinney, Bostow, Laws, Blumenfeld, and Hopkins, 1971; 
Olsen, Weber and Dorner, 1968; Standlee and Pashan, 1960). While there is not an abundant 
amount of research on quizzing in higher education, it has not gone completely unnoticed either. 
Studies completed by Anderson (1984), Burns and Vinchur (1992), Connor-Greene (2000), 
Hagen (2000), and Peckham and Roe (1977) demonstrate the consistent and equivocal findings, 
over time, of the research on quizzing in higher education. These authors, as a whole or in 
separate, suggest that quizzing has demonstrable impact on student engagement and student 
perception toward learning, yet, is not independently linked to student learning / performance. 

Studies on traditional in-class quizzing have served to inform faculty of the effectiveness 
of various teaching and assessment practices on student engagement, performance, and 
perception. For example, some studies suggest that in-class quizzing leads to an increase in the 
amount of time students spend reading course material out-of class (Connor-Greene, 2000; 
Mawhinney et al. 1971). This finding is especially encouraging as out-of-class reading by a 
student tends to promote academically engaged time (paying attention, contributing to 
discussions, and answering questions) by the student during the class (Ehrlich, 1995). Moreover, 
this academically engaged time is second only to general ability regarding having documented 
positive effects on classroom achievement or academic performance (Berliner, 1979; Denham 
and Lieberman, 1980). 

In addition, according to Barbarick (1998), Connor-Greene (2000), Crooks (1988), 
Ehrlich (1995), and Hagen (2000), when quizzing is used in a course, students perceive 
themselves as being more prepared for class meetings and more prepared for class assessments. 
This increase in self-perception may motivate the learners to participate in class more by adding 
to discussions and asking questions (Connor-Greene, 2000). 

However, with regard to academic performance, the findings have been more balanced. 
Anderson (1984), Connor-Greene (2000), and Olsen et al. (1968) offer views that, while 
admittedly not highly generalized, indicate that quizzing does not automatically lead to higher 
test scores or increases in student performance as indicated by final grades. This is in contrast to 
Barbarick (1998), Beaulieu and Zar (1986), Crooks (1988), Hagen (2000), and Martin and 
Srikameswaran (1974) who suggests that, with appropriate content overlap between the quizzes 
and performance assessments (tests), quizzes may lead to higher student performance on tests or 
in the form of final grades. 

What is novel, however, is utilizing the current technology found in higher education 
today to deliver the quizzes. More specifically, college faculty has at their disposal today 
technologies and expertise that was simply not available forty years ago. As a result, when 
considering quizzing as an assessment tool, a teaching technique, or some combination, faculty 
currently can construct and deliver quizzing which looks substantially different to that found in 
college classrooms thirty, twenty, or even ten years ago. As a result, the use of out-of-class and 
on-line quizzes by instructors in higher education and their effect on student engagement, 
learning, and perception warrants further investigation. 
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Therefore, the theoretical framework adopted for this study views student academic 
indicators, first and foremost, as being able to be influenced by the classroom instructor. 
Secondly, that the student academic indicators are engagement, performance, and perception. 
This view is consistent with the theory as presented by Berliner (1979) and Denham and 
Lieberman (1980). 

III. Method. 

A. Subjects. 

Subjects (N=24) were enrolled in a required sophomore-level functional anatomy course. 
Students were informed on the course syllabus and during the first meeting day about their 
participation in the study. 

It should be noted that this course is required for the enrolled students. As a result, there 
was a concern that manipulating the graded assignments and offering the course in a different 
way than how it is normally offered may unnecessarily add stress to the enrolled students. This 
course is normally delivered with daily in-class quizzes over the entire semester. For this study 
we gave quizzes for only one-half of a semester and made them out-of-class and on-line. The 
absolute impact of the quizzes as a course grade assignment was similar in both instances 
however. In the traditional course offering the value of the quizzes is 33% of the student’s total 
grade and for this study the quizzes equaled 24% of the students’ total grade. 

B. Materials. 

The on-line quizzes were generated by the course instructor and made available to the 
students using the university portal known as ONCOURSE tm . All registered university students 
have access to this medium and when a student accesses the site their log-on and log-off times 
are recorded. Moreover, to decrease the chance of cheating on the quizzes there is a time limit to 
finish the quiz. 

In addition, the students needed to complete the quiz prior to the corresponding lecture 
and could do so no earlier than 72 hours before class time and no later than 30 minutes prior to 
class. Moreover, the content of the quizzes directly reflected the assigned reading. 

C. Definitions. 

Student Classroom Engagement-. Engagement was primarily defined by the student 
interactions that occurred during class (questions asked and questions answered). The first way 
we recorded student interactions was by a simple percentage of the class that asked a question to 
the instructor during the lecture sessions. Second, we assessed the total number of questions 
generated by the students to the instructor during the lecture sessions. Finally, we compressed 
Bloom’s (1956) taxonomy into the three categories of Knowledge, Comprehension, and 
Application and evaluated the level of each question that students asked to the instructor. This 
compression is similar to the use of Bloom’s (1956) simplified taxonomy by Cox and Clark 
(1998) and Crooks (1988). We trained a recorder to code every student-generated question 
directed to the instructor. If the question had a “recall” aspect to it was coded as a “Knowledge” 
question; if it had an “understanding or translation” aspect then it was coded as a 
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“Comprehension” question; and, if it contained a “use or application” aspect then it was coded as 
an “Application” question. Herein, these are known as K, C, and A, respectively. 

Each course meeting the trained recorder was present and tallied the student interactions 
as they occurred. However, to ensure accuracy we videotaped each course meeting and the 
trained recorder would validate their recordings by reviewing the videotape. Again, this was 
completed for both the quizzed and non-quizzed parts of the semester. 

Regarding questions answered by the student, the instructor posed three questions to the 
entire class at the conclusion of the class session. There was one question at the K level, one at 
the A level, and one at the C level. Students were made to respond via paper/pencil and turn in 
their paper at the end of class. The responses were evaluated and recorded on a scale of 0-3. If a 
student was present yet did not / could not answer the questions it was recorded as a zero, if they 
answered 1 correct, then that equaled a 1, etc. If a student was absent it was recorded as 
“missing”. An individual running average was recorded as was a daily class average. However, it 
should be noted that the students were fully aware these scores had NO impact on their course 
grade. 

Finally, attendance was used as an indicator of student engagement. Attendance was 
tracked throughout the entire semester and stratified into “no-quiz” attendance and “quiz” 
attendance. 

Academic Performance: A students’ academic performance was evaluated by multiple 
choice tests completed by paper and pencil. Tests were administered at three points in the 
semester (pre, mid-term, and final). The pre-test contained 20 questions and was given the very 
first class meeting. The mid-term test was administered at the halfway point of the semester and 
the final test was administered during the last class meeting. 

Ten of the pretest questions were reflective of material to be covered during the first half 
of the semester and were then embedded in the mid-term test. Similarly, the second ten questions 
of the pre-test were reflective of material to be covered during the second half of the semester 
and were then embedded on the final exam. 

Again, each respective bank of ten questions came directly from the related reading and 
also varied with respect to the taxonomies used: K, C, and A. Furthermore, these questions were 
selected from questions generated from the previous semesters’ test question analysis. Only 
questions associated with an index of discrimination of 40-60% were used for the pre-test, and 
subsequent mid- and final tests. 

Student Perceptions: Student perceptions of their own level of engagement in the course 
were assessed following the first half of the course and, again, following the second half of the 
course. The instrument used was adapted from a university- specific student engagement project. 
Differences in perception were compared and reported as both individual and class average 
changes in class perception from the quiz and no-quiz portions of the course. 

D. Procedures. 

The first half of the semester (7.5 weeks) was dedicated as the no-quiz portion of the 
course and the second half of the semester (7.5 weeks) employed on-line out-of-class quizzing. 
To ensure consistency throughout the semester the reading schedule and class structure were 
controlled. 
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The reading schedule was formatted the same on the syllabus for the entire semester. 
Students clearly knew what reading assignment was to be completed prior to attending class 
throughout the entire semester. 

The instructor structured the class sessions the same throughout the semester. The 
instructor started each class with a review of the reading and a chance to ask questions, then 
proceeded with the lecture (fielding questions throughout), and ending with the “informal quiz”. 

The pre-test was administered the first day of class. At the mid-point of the semester the 
students were given the mid-term test and the self-perception inventory. 

Starting at the eighth week of the semester, the students were reminded that graded 
quizzes were being implemented and that they needed to complete the quiz prior to coming to 
class. At the end of the semester, the students took the final test and the self-perception inventory 
was repeated. 

E. Statistical Analysis. 

A paired samples t-test and Pearson Product Moment Correlation were performed on the 
data collected for the quiz and no-quiz models. The level of significance was set at p < 0.05 for 
all analyses. 

IV. Results and Discussion. 

A. Engagement. 

Analysis revealed no statistically significant difference between the percentage of 
students asking questions during the no-quiz and the quiz portions of the course. However, the 
total number of questions asked between the no-quiz and the quiz portion of the course was 
significantly different. During the no-quiz period there were 34 total questions asked in class 
during the first-half of the course. This is opposed to 74 total questions asked in class during the 
second half of the course where quizzing was utilized. 

As such, the same “pool” of students tended to ask questions whether there was a quiz or 
not, yet the number of questions generated by this “pool” of students increased after the quizzes 
started. This increase could be explained away by arguing that students got more comfortable 
with the course or instructor over time, as such, the students asked more questions during class 
time. However, the results do not suggest this. 

If the students got used to the course and instructor over time (7.5 weeks) one would 
predict that the distribution of questions would gradually increase over this time. Where the 
fewest questions were asked during week one, when the students would have been the least 
comfortable with the course or instructor. And, the most questions would have been asked during 
week seven, when the students should have been more comfortable with the course and 
instructor. Yet, the results do not reflect this trend. Weeks two and three yielded the most K, C, 
and A questions from the students, then there was a slight decrease and stabilization of student 
generated questions from weeks four through seven. One could speculate that this question 
distribution indicates that the students were comfortable with the course and instructor by at least 
week three. 

Moreover, when factoring in the nominal increase with the percentage of students asking 
questions over the first-half of the semester, one might conclude the students were comfortable 
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with the course and instructor early on. As a result, it may very well be that on-line out-of-class 
quizzing, and the necessary pre-class reading associated to that, may lead to more engaged 
students during class time. 

Regarding the level of student questions asked, it was found that no statistically 
significant difference existed regarding the number of K level questions asked by the students 
during the no-quiz portion of the course and the quizzed portion of the course (p = 0.824). 
However, there was a statistically significant difference in the number of C and A level questions 
asked between the no-quiz and quiz portions of the course (p = 0.026 and p = 0.002, 
respectively). While this absolute and significant increase in level of question asked cannot be 
definitively linked to the introduction of the quizzes, the results do illustrate another positive 
trend toward more in-class student engagement when on-line out-of-class quizzes are introduced. 

There was also a statistically significant difference in regard to student answers of the 
instructor questions at the end of each class session. It was found that the number of “zero” or 
“present but could not answer” responses significantly decreased from the no-quiz to the quizzed 
portion of the course (p = 0.000). 

This finding supports the idea that by assigning graded quizzes there will be an increase 
in student preparation and a subsequent increase in a student’s ability to answer questions about 
the course material. More important though, this finding suggests that on-line out-of-class 
quizzes yield the same benefit as in-class quizzes but without the “expense” of class time and 
management. 

It was found that on-line out-of class quizzing had no impact on class attendance (p = 
0.68). The same students missed class whether it was the no quiz or quizzed portion of the 
course. 

B. Academic Performance. 

There was no correlation between the pre-test score (average of 2.4/10) and the mid-term 
score (average of 5.4/10) in regard to the ten-question component (r = 0.267). Moreover, there 
was no correlation between the corresponding ten-component pre-test score (average of 2.2/10) 
and the corresponding ten-component final test score (average of 4.9/10), the r = 0.231. 

Both of these results were expected and favorable as there should be a significant 
difference, and subsequent low correlation, between a pre-test score and, in essence, a post-test 
score. This implies that the “treatment” (quiz) had a positive impact. 

For the purposes of this study, however, the fact that the rates of improvement were so 
similar for both the no-quiz and quizzed portions of the course indicates that on-line quizzing 
may not automatically lead to better test scores. There are many factors that influence test-scores. 
These may include but not be limited to (a) content overlap between the reading and the test 
questions, (b) format of test questions, (c) the wording of the questions, and (d) the taxonomy of 
the test questions. Without careful control, test questions themselves could influence test scores 
more than anything, interventions included (in our case, quizzes). 

Admittedly, our criterion for test question selection may have been too restrictive and 
aggressive, thus, placing a low ceiling on how much improvement one may expect from pre-test 
to post-test. As a result, any conclusions regarding on-line quizzing and academic performance, 
via test score, are preliminary as stated here. 
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C. Student Perceptions. 

Some student perceptions changed significantly from the no-quiz portion of the course to 
the quizzed portion of the course; however, not all of them were favorable. Regarding the prompt 
“I plan specific study times for this class”, during the no-quiz portion slightly over 50% of the 
students either agreed or strongly agreed, however, during the quizzed portion slightly over 80% 
of the students either agreed or strongly agreed to that statement. This indicates a formal 
acknowledgement by the student to dedicate time to reading and studying. 

However, responses to “Attending class is critical to achieving the grade I expect in this 
class” went from 95% of the students either agreeing or strongly agreeing to that statement to 
about 80% during the no-quiz portion and quiz portion of the course, respectively. It is difficult 
to ascertain the cause for this shift, however, one could speculate that by doing the reading in 
advance for the quiz, the actual class session and related material would be more familiar and 
less novel to the student. 

Finally, 83% of the twenty-four students responding to the student perception survey felt 
that the on-line out-of-class quizzes made them “learn the material better”. Moreover, 80% of the 
respondents felt that graded on-line out-of-class quizzes motivated them to “read prior to coming 
to class”, and, finally, 71% of the students felt the on-line out-of-class quizzes “got them ready 
for class discussions”. It is clear that student perception of on-line out-of-class quizzing was 
favorable. 

V. Conclusions. 

The results of this initial investigation are encouraging. We have concluded the 
following: 

1. The hypothesis statement: “On-line quizzes would lead to an increase in student 
classroom engagement” is partially supported. Specifically: 

a. The total number of questions asked during class by students increased 
significantly as a result of on-line quizzing. 

b. The C and A level questions asked by the students increased significantly as a 
result of the on-line quizzing. 

c. Students answered the instructor questions at a significantly greater rate during 
the quizzed portion of the course than the no-quiz portion. 

d. The percentage of students who asked questions during class did not increase 
during the quizzed portion of the course. 

e. The K level questions asked by the students did not increase as a result of on-line 
quizzing. 

f. On-line quizzing did not influence class attendance. 

2. The hypothesis statement: “On-line quizzes would lead to an increase in academic 
performance” was not supported. 

3. The hypothesis statement: “On-line quizzes would lead to an increase in student 
perception” was partially supported. 

a. There were four prompts that students positively attributed to on-line quizzes and 
one prompt which students negatively attributed to on-line quizzes. 
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VI. Reflective Critique. 

As a result of completing this preliminary study, the authors have continued their 
professional dialogue on the merits of quizzing, generally, and out-of-class on-line quizzing, 
specifically. While each of our areas of expertise and individual pedagogical beliefs drive our 
instructional and assessment methods, this research project has undoubtedly influenced all of us. 
The findings from this study have confirmed some of our intuitions while at the same time 
tempered other intuitions. Moving forward, we (collectively and individually) have confidence 
of what quizzing may or may not accomplish as part of a college classroom. This perspective 
will help guide the development of instructional strategies to help improve the teaching and 
learning process. 

In addition, we generally agree that on-line out-of-class quizzing is an attractive 
alternative to traditional in-class quizzing that, at best, contributes to the improvement of certain 
student academic indicators. As important, we generally agree that, at worst, on-line out-of-class 
quizzing does no harm to the student academic indicators. Given the unintended benefits of out- 
of-class on-line quizzing, such as (a) timely feedback to the student (b) automatic grading and 
corresponding ease of entry into an electronic grade book and (c) saving in-class instructional 
and discussion time, we feel that out-of-class on-line quizzing can be a significant tool for 
educators to utilize. 
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